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The enclosed substitute written description conforms the patent application 
written description to US patent practice and improves grammatical usage in the text. No 
new matter is presented. The applicant requests that the substitute written description be 
used as the basis for examination of the application. 

Claim 12 has been amended to remove a lack of antecedent basis recited in the 

claim. 

New claim 25 is presented herewith for examination. 



Respectfully submitted, 



KNOPLIOCH ET AL. 
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AMENDED CLAIMS 

12 (amended). The method according to claim 11, wherein a two- 

dimensional recentering of both MRI and scanner images is carried out by means of at least 
one rotation and/or translation operation, so that a pixel of the scanner image of coordinates 
(x,y) and a pixel of the MRI image of the same coordinates (x,y) represent the same portion 
of the [organ X-rayed] object . 

25 (new) A method for combining first and second radiographic images of an 
object to provide a third image comprising the steps of: 
providing the first image by CT scanning; 
providing the second image by MR scanning; 

recentering the first and second images by a two-dimensional rotation and/or 
translation so that the coordinates of the CT image and the MR image represent the same 
portion of the object; 

fixing in the CT image gray scale levels corresponding to upper and lower limits of 
the CT interval; 

fixing in the MR image gray levels corresponding to upper and lower limits of the 
MR interval; 

combining the first and second images to provide the third image by linear 
interpolation by integration the respective lower and upper limits of the CT interval and the 
respective lower and upper limts of the MR image, the third image having gray levels which 
lie within the CT interval are replaced by pixels of the same coordinates in the MR image. 
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12. The method according to claim 11, wherein a two-dimensional 

recentering of both MRI and scanner images is carried out by means of at least one rotation 
and/or translation operation, so that a pixel of the scanner image of coordinates (x ? y) and a 
pixel of the MRI image of the same coordinates (x,y) represent the same portion of the 
object. 
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METHOD AND SYSTEM OF FUSION OF TWO DIGITAL 

RADIOGRAPHIC IMAGES 

BACKGROUND OF THE INVENTION 

The invention concerns the fusion of two digital images of an object, the 
first image of which favors a particular constituent of the object, while the 
second image favors another. 

It has a particularly important application in the medical field, in which 
a first image of a body organ obtained by scanning is fused with a second image 
of the same organ obtained by magnetic resonance imaging (MRI). 

In fact, an image obtained by means of a scanner particularly reveals the 
bony part. In such an image, the bony part is white and all the other parts, 
especially, the soft tissues, are of a homogeneous gray without contrast. On the 
other hand, an image obtained by means of MRI reveals the soft tissues in 
different shades of gray levels and the other parts like the bony parts and 
empty space are black. 

In general, in the medical field a scanner image is fused with an MRI 
image by integrating the pixels of the bony parts of the scanner image in the 
MRI image. 

The scanner images possess an absolute scale of gray levels, that is, all 
the scanner images are compatible with one another, in the sense that a given 
gray level always represents a particular organ. This absolute scale is the 
Hounsfield scale, composed of positive and negative numbers, in which the 0 
level is the gray level of water. 

An MRI image does not possess an absolute scale. The gray levels 
depend on the patient and on the image acquisition conditions. Therefore, from 
one MRI image to another, the muscle, for example, as soft tissue, is not 
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represented by the same gray level. Thus, fusion of an MRI image with a 
scanner image results in a final image whose scale is not absolute. 

In other words, the fusion of an image possessing an absolute scale with 
another image not possessing an absolutely scale results in a final image not 
possessing any absolute scale. 

Furthermore, an image not possessing any absolute scale cannot be used 
by any of the current scanner image processing software. In fact, all of that 
software uses a standard gray level format, which is the Hounsfield scale. 
Thus, a final image originating from the fusion of both scanner and MRI images 
is incompatible with any scanner image processing software. It is necessary to 
develop specific image processing software not calibrated on the Hounsfield 
scale, in order to be able to use the final image. 

BRIEF SUMMARY OF THE INVENTION 

An embodiment of the invention provides a solution to that problem by 
a scaling of the gray levels of the MRI image, in order to render the final image 
compatible with all scanner image processing software. In other words, the 
final image will be calibrated on the Hounsfield scale. 

It is an advantage of an embodiment of the invention to reduce the cost 
of investment in the development of specific software, if it is desired to carry 
out digital processing on the final image. 

An embodiment the invention uses a final image of as an image source 
for standard radiotherapy software, which is not the case with the fused images 
in the present state of the art. 

An embodiment of the invention proposes a method of fusion of a first 
digital radiographic image obtained as a result of scanning with a second 
digital radiographic image obtained by magnetic resonance imaging (MRI). 



14XZ000074 



In an embodiment of the invention, a CT interval of gray levels is 
selected in the scanner image and each pixel of the scanner image having a gray 
level lying within the CT interval is replaced by a pixel obtained by digital 
processing of the pixel of the same coordinates as the MRI image. The final 
5 image therefore corresponds to the scanner image in which the pixels of gray 
levels lying within the CT interval are thus modified. 

Furthermore, with a view to effective digital processing, a two- 
dimensional recentering of both MRI and scanner images may be carried out by 
means of at least one rotation and /or translation operation, so that a pixel of the 
10 scanner image of coordinates (x,y) and a pixel of the MRI image of the same 
coordinates (x,y) represent the same portion of the organ X-rayed. 

In other words, the range of gray levels corresponding to the soft tissues 
is replaced by a new range of gray levels. The values of the gray levels of that 
new range are obtained from an algorithm introducing certain gray levels of the 
15 MRI image. For a given pixel of the CT interval in the scanner image, the 
algorithm calculates the gray level value of the new pixel from a pixel of the 
MRI image having the same coordinates as the pixel of the CT interval having 
to be replaced. 

In an embodiment of the invention, the upper limit Bct of the CT interval 
20 is fixed at a gray level value on the Hounsfield scale, the gray level 
corresponding to the highest value of the gray levels representing the soft 
tissues visualized on the scanner image. The lower limit Act of the CT interval 
is fixed at a gray level value on the Hounsfield scale, the gray level 
corresponding to the lowest value of the gray levels representing soft tissues 
2 5 visualized on the scanner image. 

More precisely, two thresholds are fixed, defining the CT interval 
corresponding to the soft tissues in the scanner image. 
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In practice, Bct is fixed as the highest value of the soft tissues in the 
scanner image and Act is fixed as the lowest value of the soft tissues in the 
scanner image. 

The interval thus selected is an interval included in the Hounsfield scale, 
5 since the scanner image is calibrated on that scale. 

In general, in an embodiment of the invention, one selects an MR 
interval of gray levels in the MRI image, whose upper limit Bmr corresponds to 
a gray level above which the pixels are white, and whose lower limit Amr 
corresponds to a gray level below which the pixels are black. 

10 In other words, that interval takes into account all of the variation of 

gray levels in the MRI image. This variation, this contrast, represents the useful 
information on the soft tissues. 

There are then two intervals, a first CT interval in the scanner image 
included in the Hounsfield scale and a second MR interval in the MRI image 
15 not linked to the Hounsfield scale. These two intervals represent a framing of 
the soft tissues. 

In an embodiment of the invention, the digital processing consists of a 
linear interpolation by means of an affine function integrating the value of the 
lower limit Act and upper limit Bct of the CT interval in the scanner image and 
2 o the value of the lower limit Amr and upper limit Bmr of the MR interval in the 
MRI image. 

Carrying out a linear interpolation makes it possible to respect the choice 
of contrast in the MRI image. 

Preferably, for a scanner pixel having a gray level Vct lying within the 

25 CT interval, the gray level Vmr of the pixel of the same coordinates in the MRI 

image is determined, and then a gray level in the CT interval is determined 

from the affine function and from the level Vmr. The gray level Vout of each 

pixel of the final image can then be obtained by the following algorithm: 

4 
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- if Vct < Act, then 

1) Vout = Vct, 

- if Vct > Bct, then 

2) Vout - Vct, 

- if Act < Vct < Bct, then 

3) Vout = Act + (Bct - Act) (Vmr - Amr)/(Bmr - Amr). 

In other words, while maintaining the resolution of the MRI image, the 
MRI image is scaled so that the black level Amr of the MRI image corresponds 
to the lowest value Act of the soft tissues in the scanner image. Likewise, the 
white level Bmr of the MRI image corresponds to the highest value Bmr of the 
soft tissues in the scanner image. 

In fact: 

- for Vmr = Bmr, highest gray level in the MR interval of the MRI image, 

one obtains by 3) Vout = Bct, highest gray level in the CT interval of the 
scanner image, 

- and for Vmr = Amr, lowest gray level in the MR interval of the MRI 

image, 

one obtains by 3) Vout = Act, lowest gray level in the CT interval of the 
scanner image. 

Scaling causes the MR interval not calibrated on the Hounsfield scale of 
the MRI image to undergo a digital processing which makes it correspond to 
the CT interval lying within the Hounsfield interval. 
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Thus, all the gray level values Vout of the final image will be contained 
in the Hounsfield scale, which is the standard scale of scanner image 
processing. 

The invention is also directed to a system of fusion of a first digital 
5 radiographic image obtained by scanning with a second digital radiographic 
image obtained by MRI, comprising: 

means for reading pixels of the scanner image, the gray levels of which 
lie within a predetermined CT interval; 

means for reading pixels of the MRI image, the coordinates of which are 
10 identical to those of the pixels of the CT interval of the scanner image; and 

means for calculation of a third image composed of the scanner image in 
which the pixels whose gray levels lie within the CT interval are replaced by 
pixels obtained by digital processing of the pixels of the same coordinates as the 
MRI image in order to obtain an image making possible visualization of the soft 
1 5 tissues and bony tissues. 

The final image obtained is of the scanner type. It can therefore be 
processed by standard software such as Advantage Sim or even Advantage 
Windows 3D Viewer, which is not the case with the fusion images of the prior 
art. The methods of the prior art require the use of specific software in order to 
20 be able to process their fusion images. 

Other advantages and characteristics of the invention will appear on 
examination of the detailed specification of a nonlimitative embodiment and of 
the attached drawings, in which: 

BRIEF DESCRIPTION OF THE DRAWINGS 

2 5 - Figure 1 is a flow chart of an embodiment of the method according to 

the invention; 
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- Figure 2 schematically illustrates two images obtained by a 
tomography system. 

DETAILED DESCRIPTION OF THE INVENTION 

Referring in particular to the three Figures 1, 3 and 4, first of all, the two 
5 digital images are acquired. Acquisition 1 makes it possible to obtain a scanner 
image 11 illustrated in Figure 3. This image represents a view of a patient's 
head along a given plane. A part outside the head can be distinguished there, 
that is, the air 12 represented in black. The white zone 14 corresponds to the 
bony tissues and all the grayish zones 13 correspond to the soft tissues. The 
10 scanner image 11 is of particular interest because it favors visualization of the 
bony tissue. Its principal characteristic is therefore a perfect display of the bony 
tissues 14. On the other hand, the grayish zones 13 have a poor resolution, so 
that it is impossible to distinguish the contrasts in the soft tissues. 

Acquisition 2 makes it possible to obtain an image 15 by magnetic 
15 resonance imaging. It represents a view of a patient's head along the same 
plane as the scanner image 11. A black part 16 can also be distinguished there, 
corresponding to the air all around the head. Inside the head, the black zones 
17 correspond to the bony tissues and to any element other than the soft tissues, 
like air, for example. The grayish zones 18 represent the soft tissues. The 
20 principal characteristic of that image is the display of soft tissues. The 
resolution is sufficient to distinguish contrasts, elements of interest 19. On the 
other hand, it is difficult to delimit the bony parts 17, for they are merged with 
the air and every other element appearing in black on the image 15. 

These two images 11 and 15 originating from two different methods of 
2 5 acquisition 1 and 2 represent a view of the head along a given cutting plane. 

Thus, a two-dimensional recentering 3 is made in order to render the two 
images 11 and 15 superposable. For this purpose, rotation and/or translation 
operations are possibly carried out. There are tools known to the expert that 
make it possible to control the recentering operation 3. One can mention, 
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notably, the tool using a pointer, that is, a particular element is pointed to on 
the scanner image 11, for example, and a cursor appears on the same particular 
element on the MRI image 15. The same idea is exploited in the tool using a 
magnifier. 

Once the two images 11 and 15 are recentered, one determines in stage 4 
the lowest value Act of the soft tissues in the scanner image, for example, -130, 
which is a low value of soft tissues in the Hounsfield scale. One also determines 
the highest value Bct of soft tissues in the scanner image, for example, 80, which 
is a high value of soft tissues in the Hounsfield scale. These two values are both 
CT numbers. A CT number is defined from the attenuation coefficient of the 
tissue considered and from the attenuation coefficient of water: 

CT Number = X 1000 

with |Liw: attenuation coefficient of water 

(Lto: attenuation coefficient of the tissue considered. 
The CT number is expressed in Hounsfield unit. 
Table of CT numbers 

ELEMENTS OF THE HUMAN BODY CT NUMBER 



BONE (CORTEX) > 250 

BONE (MARROW) 130 ± 100 

COAGULATED BLOOD 80 ± 10 

THYROID GLAND 70 ± 10 

LIVER 50 ± 10 

MUSCLE 45 ±5 
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BLOOD 



40 ±10 



BRAIN (WHITE MATTER) 



35 ±5 



KIDNEY 



30 ±10 



BRAIN (GRAY MATTER) 



25 ±5 



FATTY TISSUE 



-100 ± 10 



Two values Amr and Bmr on the MRI image are also determined in 
stage 5. Amr is a gray level such that the lower gray levels are considered black. 
Bmr is a gray level such that the higher gray levels are considered white. 

One then proceeds with an algorithm 6 making possible scaling of the 
MR interval. According to a preferred embodiment of the invention, algorithm 
6 is applied in accordance with Figure 2. A target pixel of the scanner image of 
gray level equal to Vct is taken in the course of stage 6a. In the first place, it is 
going to be determined whether that value is included in the CT interval. For 
that purpose, both values of the lower limit Act and upper limit Bct are 
introduced. First of all, the value Vct is compared to value Act in the course of 
stage 6b. If the gray level of the target pixel Vct is less than Act, then the target 
pixel is outside the CT interval and it can then correspond to the bony tissue 14 
or to the black background 12 of the scanner image. In that case, the target 
pixel maintains its value Vct on the final image 20. 

Otherwise, if the gray level Vct is higher than Act, it is compared in the 
course of stage 6c to value Bct. If the gray level Vct is higher than the upper 
limit Bct of the CT interval, then the target pixel maintains its value Vct on the 
final image 20 in the course of stage 6f, that is, Vout, the gray level of the target 
pixel on the final image 20, is equal to Vct. Thus, for a value Vct lower than Act 
or higher than Bct, the level Vct is maintained as gray level Vout of the final 
image 20. 
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On the other hand, if Vcr is higher than Act and lower than Bct, the gray 
level Vmr of a pixel of the MRI image of the same coordinates as the target pixel 
of the scanner image is then determined in the course of stage 6d. It is then 
made to undergo a linear inter-polation at that gray level Vmr by introducing 
levels Act, Bct, Amr and Bmr. A new value Vout independent of Vcr is then 
obtained in the course of stage 6e. 

Algorithm 6 is presented in that case in the form: 

-if Vct<-130, then 

1) Vout = Vcr, 

- if Vct > 80, then 

2) Vout = Vct, 

- if -130 < Vct < 80, then 

3) Vout = -130 + (80 + 130) (Vmr - Amr) /(Bmr - Amr). 

Whatever the values Amr and Bmr, the final image 20 presents a range of 
gray levels according to the Hounsfield scale. Figure 5 shows the final image 20 
in which the bony tissues 14 as well as the soft tissues 18 are distinguished. The 
background of the image 12 remains black, as on the scanner image 11. 

However, for a scanner image obtained according to a view of the lungs, 
a pixel of gray level Vcr representing the lungs in the scanner image will have 
value Vout equal to Vcr in the final image, whatever the gray level Vct 
included or not in the CT interval. In other words, if the gray levels of the lungs 
in the final image are the gray levels of the lungs in the scanner image, the 
linear interpolation is not applied on the gray levels of the lungs. This is due to 
the fact that the gray levels representing the lungs possess such dynamics that 
the scanner image has a better resolution than the MRI image. 
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The final image 20, whose gray levels are contained in the Hounsfield 
scale, was thus determined. This final image is then safeguarded in stage 7 in 
the form of a scanner image. It can be printed in stage 10 or even displayed in 
stage 8 on a screen for possible study. But the main advantage of that methods 
resides in the fact that this image can be delivered on entry of a standard 
radiotherapy treatment system in stage 9. 

The final image originating from fusion of a scanner image with an MRI 
image reveals soft tissues as well as bony tissues and can be used a source for 
all standard scanner image processing software, such as Advantage Sim, Isis or 
even Advantage Windows Viewer. 

Various modifications in structure and/or steps and/or function may 
be made by one skilled in the art without departing from the scope of the 
invention. 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Application of : CT/MRI FUSED IMAGE 

KNOPLIOCH ET AL PRESENTATION ALGORITHM 

Serial No.: : Group Art Unit: 

Filed: April 28, 1999 : Examiner: 

December^, 2000 



PRELIMINARY AMENDMENT 

BOX PCT 

Assistant Commissioner for Patents 
Washington, D.C. 20231 

Sir: 

Before calculation of the official filing fee and before examination of the 
application please amend the application as follows: 
IN THE ABSTRACT: 

Delete the following text: " Reference : no figure" 

"[Keys to text in Fig. 2:J " 
"OUf= YES" 
-NON = NO" 

IN THE TITLE: Change the title to read: -METHOD AND SYSTEM OF 
FUSION OF TWO DIGITAL RADIOGRAPHIC IMAGES - 

IN THE CLAIMS: 

Cancel claims 1 to 10 and add new claims 1 1 to 24, as follows: 

11. A method of fusion of first and second digital radiographic images of an 
object comprising the steps of: 

scanning the object to provide first digital radiographic image; 

provided the second digital radiographic image by magnetic resonance imaging 
(MRI); and 
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selecting in the scanner image a CT interval of gray wherein each pixel of the 
scanner image having a gray level lying within the CT interval is replaced by a 
pixel obtained by digital processing of the pixel of the same coordinates as the MRI image, 
the final image corresponding to the scanner image in which the pixels of gray levels lying 
within the CT interval are thus modified 

12. The method according to claim 11, wherein a two-dimensional recentering 
of both MRI and scanner images is carried out by means of at least one rotation and/or 
translation operation, so that a pixel of the scanner image of coordinates (x,y) and a pixel of 
the MRI image of the same coordinates (x,y) represent the same portion of the organ X- 
rayed. 

13. The method according to claim 11 wherein the upper limit B CT of the CT 
interval is fixed at a gray level value on the Hounsfield scale, the gray level corresponding to 
the highest value of the gray levels representing the soft tissues visualized on the scanner 
image. 

14. The method according to claim 12 wherein the upper limit Bct of the CT 
interval is fixed at a gray level value on the Hounsfield scale, the gray level corresponding to 
the highest value of the gray levels representing the soft tissues visualized on the scanner 
image. 

15. The method according to claim 11 wherein the lower limit Act of the CT 
interval is fixed at a gray level value on the Hounsfield scale, the gray level corresponding to 
the lowest value of the gray levels representing soft tissues visualized on the scanner image. 

16. The method according to claim 12 wherein the lower limit Act of the CT 
interval is fixed at a gray level value on the Hounsfield scale, the gray level corresponding to 
the lowest value of the gray levels representing soft tissues visualized on the scanner image. 

17. The method according to claim 13 wherein the lower limit Act of the CT 
interval is fixed at a gray level value on the Hounsfield scale, the gray level corresponding to 
the lowest value of the gray levels representing soft tissues visualized on the scanner image. 



18. The method according to claim 14 wherein the lower limit Act of the CT 
interval is fixed at a gray level value on the Hounsfield scale, the gray level corresponding to 
the lowest value of the gray levels representing soft tissues visualized on the scanner image. 

19. The method according to claim 11 wherein one selects another MR interval 
of gray levels in the MRI image, whose upper limit Bmr corresponds to a gray level above 
which the pixels are white. 

20. The method according to claim 19 wherein the lower limit Amr of the MR 
interval corresponds to a gray level below which the pixels are black. 

21. The method according to claim 1 1 wherein the digital processing consists of 
a linear interpolation. 

22. The method according to claim 21 wherein the linear interpolation 
introduces an affine function integrating the value of the lower limit Act and upper limit Bct 
of the CT interval in the scanner image and the value of the lower limit Amr and upper limit 
Bmr of the MR interval in the MRI image. 

23. The method according to claim 22 wherein a scanner pixel having a gray 
level Vct lying within the CT interval, the gray level Vmr of the corresponding pixel in the 
MRI image is determined, and then a gray level in the CT interval is determined from the 
affine function and from the level Vmr; the gray level V 0 ut of each pixel of the final image 
is then obtained by the following algorithm: 

-ifV C T< Act, then 

1) Vout = Vct, 
-ifVcT> Bct, then 

2) Vout = Vct, 

- if Act < Vct < B C t, then 

3) Vqut = A CT + (B C t - A CT ) (V mr - Amr)/(B M r - Amr). 



24. A system of fusion of first and second digital radiographic images 
comprising: 

means for providing the first digital radiographic image by scanning; 

means for providing the second digital radiographic image by MRI; 

means for reading pixels of the scanner image, the gray levels of which lie 
within a predetermined CT interval; 

means for reading pixels of the MRI image, the coordinates of which are 
identical to those of the pixels of the CT interval of the scanner image; and 

means for calculation of a third image composed of the scanner image in 
which the pixels whose gray levels lie within the CT interval are replaced by pixels 
obtained by digital processing of the pixels of the same coordinates as the MRI image in 
order to obtain an image making possible visualization of the soft tissues and bony 
tissues. 

REMARKS 

This amendment formats the title, abstract, and claims in conformity with U.S. 
patent practice. 

Respectfully submitted, 
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METHOD AND SYSTEM OF FUSION OF TWO DIGITAL 

BACKGROUND OF THE INVENTION 

The invention concerns the fusion of two digital images of an object, the 
first image of which favors a particular constituent of the object, while the 
second image favors another. 

It has a particularly important application in the medical field, in which 
5 a first image of a body organ obtained by scanning is fused with a second image 
of the same organ obtained by magnetic resonance imaging (MRI). 

In fact, an image obtained by means of a scanner particularly reveals the 
bony part. In such an image, the bony part is white and all the other parts, 
especially, the soft tissues, are of a homogeneous gray without contrast. On the 
10 other hand, an image obtained by means of MRI reveals the soft tissues in 
different shades of gray levels and the other parts like the bony parts and 
empty space are black. 

In general, in the medical field a scanner image is fused with an MRI 
image by integrating the pixels of the bony parts of the scanner image in the 
15 MRI image. 

The scanner images possess an absolute scale of gray levels, that is, all 
the scanner images are compatible with one another, in the sense that a given 
gray level always represents a particular organ. This absolute scale is the 
Hounsfield scale, composed of positive and negative numbers, in which the 0 
2 o level is the gray level of water. 

An MRI image does not possess an absolute scale. The gray levels 
depend on the patient and on the image acquisition conditions, Therefore, from 
one MRI image to another, the muscle, for example, as soft tissue, is not 
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represented by the same gray level Thus, fusion of an MRI image with a 
scanner image results in a final image whose scale is not absolute. 

In other words, the fusion of an image possessing an absolute scale with 
another image not possessing an absolutely scale results in a final image not 
possessing any absolute scale. 

Furthermore, an image not possessing any absolute scale cannot be used 
by any of the current scanner image processing software. In fact, all of that 
software uses a standard gray level format, which is the Hounsfield scale. 
Thus, a final image originating from the fusion of both scanner and MRI images 
is incompatible with any scanner image processing software. It is necessary to 
develop specific image processing software not calibrated on the Hounsfield 
scale, in order to be able to use said final image. 

The invention is aimed at introducing a solution to that problem by a 
scaling of the gray levels of the MRI image, in order to render the final image 
compatible with all scanner image processing software. In other words, the 
final image will be calibrated on the Hounsfield scale. 

One object of the invention is to reduce the cost of investment in the 
development of specific software, if it is desired to carry out digital processing 
on said final image. 

Another object of the invention is to use a final image of fusion obtained 
according to the method of the invention as image source for the standard 
radiotherapy software, which is not the case with the fused images in the 
present state of the art. 

The invention therefore proposes a method of fusion of a first digital 
radiographic image obtained as a result of scanning with a second digital 
radiographic image obtained by magnetic resonance imaging (MRI). 

According to a general characteristic of the invention, a CT interval of 
gray levels is selected in the scanner image and each pixel of said scanner image 
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having a gray level lying within the CT interval is replaced by a pixel obtained 
by digital processing of the pixel of the same coordinates as the MRI image. 
The final image therefore corresponds to the scanner image in which the pixels 
of gray levels lying within the CT interval are thus modified. 

5 Furthermore, with a view to effective digital processing, a two- 

dimensional recentering of both MRI and scanner images is carried out by 
means of at least one rotation and /or translation operation, so that a pixel of the 
scanner image of coordinates (x,y) and a pixel of the MRI image of the same 
coordinates (x,y) represent the same portion of the organ X-rayed. 

10 In other words, the range of gray levels corresponding to the soft tissues 

is replaced by a new range of gray levels. The values of the gray levels of that 
new range are obtained from an algorithm introducing certain gray levels of the 
MRI image. For a given pixel of the CT interval in the scanner image, the 
algorithm calculates the gray level value of the new pixel from a pixel of the 

15 MRI image having the same coordinates as the pixel of the CT interval having 
to be replaced. 

According to one method of use of the invention, the upper limit Bcr of 
the CT interval is fixed at a gray level value on the Hounsfield scale, said gray 
level corresponding to the highest value of the gray levels representing the soft 
20 tissues visualized on the scanner image. The lower limit Act of the CT interval 
is fixed at a gray level value on the Hounsfield scale, said gray level 
corresponding to the lowest value of the gray levels representing soft tissues 
visualized on the scanner image. 

More precisely, two thresholds are fixed, defining the CT interval 
2 5 corresponding to the soft tissues in the scanner image. 

In practice, Bcr is fixed as the highest value of the soft tissues in the 
scanner image and Act is fixed as the lowest value of the soft tissues in the 
scanner image. 
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The interval thus selected is an interval included in the Hounsfield scale, 
since the scanner image is calibrated on that scale. 

In general, according to one method of use of the invention, one selects 
an MR interval of gray levels in the MRI image, whose upper limit Bmr 
corresponds to a gray level above which the pixels are white, and whose lower 
limit Amr corresponds to a gray level below which the pixels are black. 

In other words, that interval takes into account all of the variation of 
gray levels in the MRI image. This variation, this contrast, represents the useful 
information on the soft tissues. 

There are then two intervals, a first CT interval in the scanner image 
included in the Hounsfield scale and a second MR interval in the MRI image 
not linked to the Hounsfield scale. These two intervals represent a framing of 
the soft tissues. 

According to a method of use of the invention, the digital processing 
consists of a linear interpolation by means of an affine function integrating the 
value of the lower limit Act and upper limit Bcr of the CT interval in the 
scanner image and the value of the lower limit Amr and upper limit Bmr of the 
MR interval in the MRI image. 

Carrying out a linear interpolation makes it possible to respect the choice 
of contrast in the MRI image. 

Preferably, for a scanner pixel having a gray level Vct lying within the 
CT interval, the gray level Vmr of the pixel of the same coordinates in the MRI 
image is determined, and then a gray level in the CT interval is determined 
from said affine function and from said level Vmr. The gray level Vout of each 
pixel of the final image can then be obtained by the following algorithm: 

- if Vct < Act, then 

1) Vout = Vct, 
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- if Vct > Bcr, then 

2) Vout = Vct, 

- if Act < Vct < Bcr, then 

3) Vout = Act + (Bcr - Act) (Vmr - Amr)/ (Bmr - Amr). 

In other words, while maintaining the resolution of the MRI image, the 
MRI image is scaled so that the black level Amr of the MRI image corresponds 
to the lowest value Act of the soft tissues in the scanner image. Likewise, the 
white level Bmr of the MRI image corresponds to the highest value Bmr of the 
soft tissues in the scanner image. 

In fact: 

- for Vmr = Bmr, highest gray level in the MR interval of the MRI image, 

one obtains by 3) Vout = Bcr, highest gray level in the CT interval of the 
scanner image, 

- and for Vmr = Amr, lowest gray level in the MR interval of the MRI 

image, 

one obtains by 3) Vout = Act, lowest gray level in the CT interval of the 
scanner image. 

Scaling causes the MR interval not calibrated on the Hounsfield scale of 
the MRI image to undergo a digital processing which makes it correspond to 
the CT interval lying within the Hounsfield interval. 

Thus, all the gray level values Vout of the final image will be contained 
in the Hounsfield scale, which is the standard scale of scanner image 
processing. 

The invention also concerns a system of fusion of a first digital 
radiographic image obtained by scanning with a second digital radiographic 
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image obtained by MRI, comprising: 

- a means of reading pixels of the scanner image, the gray levels of which 
lie within a predetermined CT interval, 

- a means of reading pixels of the MRI image, the coordinates of which 
are identical to those of the pixels of the CT interval of the scanner image, 

- a means of calculation of a third image composed of the scanner image 
in which the pixels whose gray levels lie within the CT interval are replaced by 
pixels obtained by digital processing of the pixels of the same coordinates as the 
MRI image in order to obtain an image making possible visualization of the soft 
tissues and bony tissues. 

The final image obtained is of the scanner type. It can therefore be 
processed by standard software such as Advantage Sim or even Advantage 
Windows 3D Viewer, which is not the case with the fusion images of the prior 
art. The methods of the prior art require the use of specific software in order to 
be able to process their fusion images. 

Other advantages and characteristics of the invention will appear on 
exa Tination of the detailed specification of a nonlimitative embodiment and of 
the attached drawings, in which: 

BRIEF DESCRIPTION OF THE DRAWINGS 

- Figure 1 is a flow chart of an embodiment of the method according to 
the invention; 

- Figure 2 schematically illustrates two images obtained by a 
tomography system. 

DETAILED DESCRIPTION OF THE INVENTION 

Referring in particular to the three Figures 1, 3 and 4, first of all, the two 
digital images are acquired. Acquisition 1 makes it possible to obtain a scanner 
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image 11 illustrated in Figure 3. This image represents a view of a patients 
head along a given plane. A part outside the head can be distinguished there, 
that is, the air 12 represented in black. The white zone 14 corresponds to the 
bony tissues and all the grayish zones 13 correspond to the soft tissues. The 
scanner image 11 is of particular interest because it favors visualization of the 
bony tissue. Its principal characteristic is therefore a perfect display of the bony 
tissues 14. On the other hand, the grayish zones 13 have a poor resolution, so 
that it is impossible to distinguish the contrasts in the soft tissues. 

Acquisition 2 makes it possible to obtain an image 15 by magnetic 
resonance imaging. It represents a view of a patient's head along the same 
plane as the scanner image 11. A black part 16 can also be distinguished there, 
corresponding to the air all around the head. Inside the head, the black zones 
17 correspond to the bony tissues and to any element other than the soft tissues, 
like air, for example. The grayish zones 18 represent the soft tissues. The 
principal characteristic of that image is the display of soft tissues. The 
resolution is sufficient to distinguish contrasts, elements of interest 19. On the 
other hand, it is difficult to delimit the bony parts 17, for they are merged with 
the air and every other element appearing in black on the image 15. 

These t images 11 and 15 originating from two different methods of 
acquisition 1 and 2 represent a view of the head along a given cutting plane. 
Thus, a two-dimensional recentering 3 is made in order to render the two 
images 11 and 15 superposable. For this purpose, rotation and/or translation 
operations are possibly carried out. There are tools known to the expert that 
make it possible to control said recentering operation 3. One can mention, 
notably, the tool using a pointer, that is, a particular element is pointed to on 
the scanner image 11, for example, and a cursor appears on the same particular 
element on the MRI image 15. The same idea is exploited in the tool using a 
magnifier. 

Once the two images 11 and 15 are recentered, one determines in stage 4 
the lowest value Act of the soft tissues in the scanner image, for example, -130, 
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which is a low value of soft tissues in the Hounsfield scale. One also determines 
the highest value Bcr of soft tissues in the scanner image, for example, 80, which 
is a high value of soft tissues in the Hounsfield scale. These two values are both 
CT numbers. A CT number is defined from the attenuation coefficient of the 
5 tissue considered and from the attenuation coefficient of water: 

CT Number = xlOOO 

with Uw: attenuation coefficient of water 

uo: attenuation coefficient of the tissue considered. 
The CT number is expressed in Hounsfield unit. 



l o Table of CT numbers 

ELEMENTS OF THE HUMAN BODY CT NUMBER 

BONE (CORTEX) > 250 

BONE (MARROW) 130 ± 100 

COAGULATED BLOOD 80 ± 10 

15 THYROID GLAND 70 ±10 

LIVER 50 ± 10 

MUSCLE 45 ± 5 

BLOOD 40 ± 10 

BRAIN (WHITE MATTER) 35 ± 5 

20 KIDNEY 30 ±10 

BRAIN (GRAY MATTER) 25 ± 5 

FATTY TISSUE -100 ± 10 
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Two values Amr and Bmr on the MRI image are also determined in 
stage 5. Amr is a gray level such that the lower gray levels are considered black. 
Bmr is a gray level such that the higher gray levels are considered white. 

One then proceeds with an algorithm 6 making possible scaling of the 
5 MR interval. According to a preferred embodiment of the invention, algorithm 
6 is applied in accordance with Figure 2. A target pixel of the scanner image of 
gray level equal to Vct is taken in the course of stage 6a. In the first place, it is 
going to be determined whether that value is included in the CT interval For 
that purpose, both values of the lower limit Act and upper limit Bct are 
10 introduced. First of all, said value Vct is compared to value Act in the course of 
stage 6b. If the gray level of the target pixel Vct is less than Act, then the target 
pixel is outside the CT interval and it can then correspond to the bony tissue 14 
or to the black background 12 of the scanner image. In that case, the target 
pixel maintains its value Vct on the final image 20. 

15 Otherwise, if the gray level Vct is higher than Act, it is compared in the 

course of stage 6c to value Bct. If the gray level Vct is higher than the upper 
limit Bct of the CT interval, tnen said target pixel maintains its value Vct on the 
final image 20 in the course of stage 6f, that is, Vout, the gray level of the target 
pixel on the final image 20, is equal tc Vct. Thus, for a value Vct lower than Act 

20 or higher than Bct, said level Vct is maintained as gray level Vout of the final 
image 20. 

On the other hand, if Vcr is higher than Act and lower than Bct, the gray 
level Vmr of a pixel of the MRI image of the same coordinates as said target 
pixel of the scanner image is then determined in the course of stage 6d. It is 
2 5 then made to undergo a linear inter-polation at that gray level Vmr by 
introducing levels Act, Bct, Amr and Bmr. A new value Vout independent of 
Vct is then obtained in the course of stage 6e. 

Algorithm 6 is presented in that case in the form: 



- if Vct < -130, then 
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1) VOUT = VCT, 

- if Vct > 80, then 

2) Vout = Vct, 

- if -130 < Vct < 80, then 

3) Vout = -130 + (80 + 130) (Vmr - Amr)/(Bmr - Amr). 

Whatever the values Amr and Bmr, the final image 20 presents a range of 
gray levels according to the Hounsfield scale. Figure 5 shows said final image 
20 in which the bony tissues 14 as well as the soft tissues 18 are distinguished. 
The background of the image 12 remains black, as on the scanner image 11. 

However, for a scanner image obtained according to a view of the lungs, 
a pixel of gray level Vct representing said lungs in the scanner image will have 
value Vout equal to Vct in the final image, whatever the gray level Vct 
included or not in the CT interval. In other words, if the gray levels of the lungs 
in the final image are the gray levels of the lungs in the scanner image, the 
linear interpolation is not applied on the gray levels of the lungs. This is due to 
the fact that the gray levels representing the lungs possess such dynamics that 
the scanner image has a better resolution than the MRI image. 

The final image 20, whose gray levels are contained in the Hounsfield 
scale, was thus determined. This final image is then safeguarded in stage 7 in 
the form of a scanner image. It can be printed in stage 10 or even displayed in 
stage 8 on a screen for possible study. But the main advantage of that methods 
resides in the fact that this image can be delivered on entry of a standard 
radiotherapy treatment system in stage 9. 

The final image originating from fusion of a scanner image with an MRI 
image reveals soft tissues as well as bony tissues and can be used a source for 
all standard scanner image processing software, such as Advantage Sim, Isis or 
even Advantage Windows Viewer. 
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Various modifications in structure and /or steps and /or function may 
be made by one skilled in the art without departing from the scope of the 
invention. 
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WHAT IS CLAIMED IS: 

1. Method of fusion of a first digital radiographic image obtained as 
a result of scanning with a second digital radiographic image obtained by 
magnetic resonance imaging (MRI), in which a CT interval of gray levels is 
selected in the scanner image and each pixel of said scanner image having a 
gray level lying within the CT interval is replaced by a pixel obtained by digital 
processing of the pixel of the same coordinates as the MRI image, the final 
image corresponding to the scanner image in which the pixels of gray levels 
lying within the CT interval are thus modified. 

2. Method according to Claim 1, characterized in that a two- 
dimensional recentering of both MRI and scanner images is carried out by 
means of at least one rotation and/or translation operation, so that a pixel of 
said scanner image of coordinates (x,y) and a pixel of the MRI image of the 
same coordinates (x,y) represent the same portion of the organ X-rayed. 

3. Method according to one of the foregoing claims, characterized in 
that the upper limit Bcr of the CT interval is fixed at a gray lev^l value on the 
Hounsfield scale, said gray level corresponding to the highest value of the gray 
levels representing the soft tissues visualized on the scanner image. 

4. Method according to one of the foregoing claims, characterized in 
that the lower limit Act of the CT interval is fixed at a gray level value on the 
Hounsfield scale, said gray level corresponding to the lowest value of the gray 
levels representing soft tissues visualized on the scanner image. 

5. Method according to one of the foregoing claims, characterized in 
that one selects another MR interval of gray levels in the MRI image, whose 
upper limit Bmr corresponds to a gray level above which the pixels are white. 

6. Method according to Claim 5, characterized in that the lower limit 
Amr of the MR interval corresponds to a gray level below which the pixels are 
black. 
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7. Method according to one of the foregoing claims, characterized in 
that the digital processing consists of a linear interpolation. 

8. Method according to Claim 7, characterized in that the linear 
interpolation introduces an affine function integrating the value of the lower 
limit Act and upper limit Bct of the CT interval in the scanner image and the 
value of the lower limit Amr and upper limit Bmr of the MR interval in the MRI 
image. 

9. Method according to Claim 8, characterized in that for a scanner 
pixel having a gray level Vct lying within the CT interval, the gray level Vmr of 
the corresponding pixel in the MRI image is determined, and then a gray level 
in the CT interval is determined from said affine function and from said level 
Vmr; the gray level Vout of each pixel of the final image is then obtained by the 
following algorithm: 

- if Vct < Act, then 

1) Vout = Vct, 

- if Vct > Bct, then 

2) Vout - Vct, 

- if Act < Vct < Bct, then 

3) Vout = Act + (Bct - Act) (Vmr - Amr)/(Bmr - Amr). 

10. System of fusion of a first digital radiographic image obtained by 
scanning with a second digital radiographic image obtained by MRI, 
characterized in that it comprises: 

- a means of reading pixels of the scanner image, the gray levels of 
which lie within a predetermined CT interval, 
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- a means of reading pixels of the MRI image, the coordinates of 
which are identical to those of the pixels of the CT interval of the scanner image, 

- a means of calculation of a third image composed of the scanner 
image in which the pixels whose gray levels lie within the CT interval are 
replaced by pixels obtained by digital processing of the pixels of the same 
coordinates as the MRI image in order to obtain an image making possible 
visualization of the soft tissues and bony tissues. 
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METHOD AND SYSTEM OF FUSION OF TWO DIGITAL 
RADIOGRAPHIC IMAGES 

ABSTRACT OF THE DISCLOSURE 

Method of fusion of a first digital radiographic image obtained as a 
result of scanning with a second digital radiographic image obtained by 
magnetic resonance imaging (MRI). 

A CT interval of gray levels is selected first in the scanner image. Each 
pixel of the scanner image having a gray level lying within the CT interval is 
then replaced by a pixel obtained by digital processing of the pixel of the same 
coordinates as the MRI image. 

Then, the final image corresponds to the scanner image in which the 
pixels of gray levels lying within the CT interval have undergone said digital 
processing. 

Reference : no figure. 

[Keys to text in Fig. 2:] 
15 0111= YES 
NON - NO 
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Ful1 name of th * rd i oint ' nventor: Jean Labarre 
Inventor's signature: y ^X*-^>^ Date:. 



£-,...3 

ff i 




Residence: 32 , rue^ezout ^ 

F-75OT4, Paris, France ^£L>C^ 
Citizenship: France 

Post Office Address: same as residence 



Full name of fourth joint inventor: 

Inventor's signature: Date:. 

Residence: 

Citizenship: 

Post Office Address: same as residence 
Full name of fifth joint inventor: 

Inventor's signature: Date: 

Residence: 

Citizenship: 

Post Office Address: same as residence 
Full name of sixth joint inventor: 

Inventor's signature: Date:_ 

Residence: 
Citizenship: 

Post Office Address: same as residence 

Full name of seventh joint inventor: 

inventor's signature: Date:„ 

Residence: 
Citizenship: 

Post Office Address: same as residence 
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